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Abstract 

This note discusses the problem of multiple testing of a single hypoth- 
esis, with a standard goal of combining a number of p-values without 
making any assumptions about their dependence structure. An old result 
by Riischendorf shows that the p-values can be combined by scaling up 
their average by a factor of 2 (but no smaller factor is sufficient in general). 



1 Introduction 

Suppose we are testing the same hypothesis using K > 2 different statistical 
tests and obtaining p-values pi, ■ ■ ■ ,Pk- How can we combine them into a single 
p- value? 

One of the earliest papers answering this question was Fisher's [2]. However, 
Fisher's paper assumes that the p-values are independent, whereas we would like 
to avoid any assumptions besides all pk, k = 1, . . . , K , being bona fide p-values. 
Fisher's method has been extended to dependent p-values in, e.g., [II [5], but 
the combined p-values obtained in those papers are approximate; in this note 
we are interested only in precise or conservative p-values. 

The simplest method for combining p-values is the Bonfcrroni method: 

F(Pi, ■ ■ ■ ,Pk) ■= K min(pi, . . . ,p K ) (1) 

(when F(pi, . . . ,pk) exceeds 1 it can be replaced by 1, but we usually ignore 
this trivial step). Albeit F(p%, . . . ,Pk) is a p- value, it has been argued that in 
many cases it is overly conservative. Riiger [IT] extends the Bonferroni method 
by showing that, for any fixed k G {1, . . . , K}, 

K 

F(pi,...,Pk) := -rP(k) (2) 

is a p- value, where p^) is the fcth smallest p- value among P\,... ,Pk', see \TU\ 
for a simpler exposition. Hommel [S] develops this by showing that 

F( P1 ,. ■ ■ lPK ) := (l + \ + ■ ■ ■ + £) L mi" K (3) 
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is also a p-value. (Simes [T3] has improved © by removing the first factor on 
the right-hand side of ([3]), but he assumes the independence of pi, . . . ,px-) 

Intuitively, the most natural way to combine K numbers is simply to average 
them; essentially, this is the way of combining p-values used in the method of 
cross-conformal prediction (see [13], (11))- None of the functions F in (JXJ) , @, 

and §S§ involves the average p :— y(piH ^Pk)- This note draws the reader's 

attention to a result by Riischendorf ([2], Theorem 1) showing that p is not 
always a p-value but 2p is; moreover, the factor of 2 cannot be improved in 
general. 

Section [2] proves the part of Riischendorf 's result stating that 2p is a bona 
fide p-value (perhaps conservative). Section [3] considers the case K = 2, in 
which it is very easy to see that the factor of 2 is optimal. 

It is often possible to automatically transform results about multiple test- 
ing of a single hypothesis into results about testing multiple hypotheses; the 
standard procedures are Marcus et al.'s [§] closed testing procedure and its 
modification by Hommel [5]. In particular, when applied to the Bonferroni 
method the closed testing procedure gives the well-known method due to Holm 
0]; see, e.g., [6] [7] for its further applications. Unfortunately, the closed test- 
ing procedure does not appear to lead to a simple and intuitive way of testing 
multiple hypotheses when combined with Riischendorf s result, and it will not 
be discussed further in this note. 

Some notation and terminology 

If E is a property of elements of a set X, 1e ■ X — > [0, oo) is the indicator 
function of E: 1e(%) = 1 if x satisfies E and 1_e(ie) = if not. A function 
F : [0,1] — > [0,oo) is increasing (resp. decreasing) if F(x\) < F{x2) (resp. 
F(xi) > F(x2)) whenever x\ < X2- A function F : [0, 1} K — > [0, oo) is increasing 
(resp. decreasing) if it is increasing (resp. decreasing) in each of its arguments. A 
set in [0, 1} K is increasing (resp. decreasing) if its indicator function is increasing 
(resp. decreasing). 

2 Combining p-values by scaled averaging 

A p-value function is a random variable P that satisfies 

F(P<e)<e, Vee[0,l]. 

The values taken by a p-value function are p-values (allowed to be conser- 
vative). (In Section Q] the expression "p-value" was loosely used to refer to 
p-value functions as well.) A merging function is an increasing Borel func- 
tion F : [0, 1} K -> [0, oo) such that F(U l: U K ) is a p-value function, where 
U\ 1 . . . ,Uk are random variables distributed uniformly on [0,1]. 

Remark. The requirement that a merging function be Borel does not follow 
automatically from the requirement that it be increasing: sec the remark after 



2 



Theorem 4.4 in [3] (Theorem 4.4 itself says that every increasing function on 
[0, l] K is Lebesgue measureable). 

Notice that, for any merging function F, F(P\, . . . , Pk) is a p- value function 
whenever Pi, ... , Pk are p- value functions. Indeed, for each k G {1, . . . ,K} we 
can define a uniformly distributed random variable Uk < Pk by 

U k (u) := P{P k < Pk{u)) +0F(Pk = P fc H), uen, 

where 9 is a random variable distributed uniformly on [0,1], and £1 is the un- 
derlying probability space extended (if required) to carry such a we then 
have 

P(F(P 1 , . . . , P K ) < e) < P(F(C/ 1 , . . . , U K ) < e) < e, Ve G [0, 1]. 

The following proposition states Riischendorf 's result in terms of merging 
functions. 

Proposition 1. The function M : [0, 1] K — ► [0, 1] defined by 

M{ Pl ,..., PK ):= ^{pi + ---+p K ) (4) 

is a merging function. 

The rest of this section is devoted to a self-contained proof of Proposition [TJ 
A copular probability measure is a probability measure on [0,1]^ all of whose 
marginals are uniform probability measures on [0, 1]. The upper copular proba- 
bility C(E) of a Borel set E C [0, 1} K is defined to be the supremum of x(E), 
x ranging over the copular probability measures. In terms of C, an increas- 
ing Borel function F : [0, l] K — > [0, oo) is a merging function if and only if 
C(F < e) < e for all e G [0, 1]. We say that a merging function F is precise if 
C(F < e) = e for all e G [0,1]. 

For s G [0, oo), define 

E s := {(%,...,%) G [0,1] A ' «! + ••• + <s} C [0,1] K . (5) 

Proposition [T] can be strengthened: in fact, M is a precise merging function. 
The original statement of this result is as follows. 

Lemma 1 ([H], Theorem 1). For any s G [0,oo) ; 



C(E S ) = min 




Remark. In Section [T] we already alluded to an example of a set with a known 
upper copular probability: the set 

{(-Ul, . . .,U K ) G [0, l] K | l Ul < a H h l« K <a > fc} , 
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where a £ [0, 1] and k € {1, . .. , K}, has upper copular probability of (K/k)a; 
this is equivalent to ([2]) being a merging function. Another well-known example 
is H := [0, tti] x ••• x [0, ujf], where ui,...,uk <E [0,1]. The upper copular 
probability of _ff is min(ui, . . . , ). This is known as one of the Frechet- 
Hoeffding bounds in the theory of copulas. Lemma [TJ is one more example of 
this kind. Lemma [5] below will give a simple characterization of upper copular 
probability in the easy case K = 2. 

Given Lemma [T] the proof of Proposition [1] is trivial: for any e S [0, 1], 
F(M(U U . . . , U K ) < e) = P (Ui + ■ ■ • + U K < ^\ < e. 

Notice that for the proof of Proposition [T] we only need the inequality < in the 
lemma. The rest of this section is devoted to the proof of this inequality. 

Let K[0, 1] be the sum of K disjoint copies of the interval [0, 1]. A (somewhat 
arbitrary) concrete representation of K[0, 1] is the set U^ =1 [2(k — 1), 2k — 1]. We 
will sometimes use the notation K[0, l]k for the kth copy of [0, 1] in K[0, 1]; so 
that K[0, l]k = [2(k — 1), 2k — 1] in the concrete representation (but K[0, l]k is 
always identified with [0, 1], via the bijection u n> u — 2(k — 1) in the concrete 
representation). If a; is a measure on [0, 1]^, we define Xk to be the projection 
of x onto the fcth coordinate of [0, 1] , 

x k (E) := x([0, if- 1 x E x [0, l] A "~ fc ), E C [0, 1] is Borel, 

and we define Ax to be the measure on K[0, 1] that coincides with Xk on K[0, l]k 
(so that Ax's total mass is K when x is a probability measure). The uniform 
measure on K[Q, 1] is the measure on the Borel cr-algebra on K[0, 1] that coin- 
cides with the uniform probability measure on each of its components K[0, l]k 
(so that Ax is the uniform measure on K[0, 1] if and only if x is a copular 
probability measure). 

Lemma [T] can be interpreted as a statement about the following infinite- 
dimensional problem of linear programming: 

cx —> sup subject to Ax = b, x > 0, (6) 

where c is the indicator function of the set E s , the variable x ranges over all 
measures on [0, 1]^, cx is understood to be J cdx, Ax is as defined above, and 
b is the uniform measure on K[0, 1], The condition x > is an embellishment 
without a formal meaning (and emphasizes the fact that measures take only 
nonnegative values). Lemma [1] says that the value of ([6]) is 2s/ K. 
The formal dual problem to © is 

Xb ->• inf subject to XA > c, (7) 

which we will interpret as follows: the dual variable A ranges over all Borel 
functions on K[0, 1], Xb is understood to be J Xdb, XA is the function on [0, 1]^ 
defined by 

(XA)(ui,. . . , u K ) ■■= Ai(«i) H h X K (u K ), 
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where Xk is the restriction of A to K[0,l]k, and > stands, as usual, for the 
pointwise inequality. 

It is easy to see that the operators x i— > Ax and A M> \A are dual, in the 
sense that (XA)x = X(Ax): 

(XA)x = J Xidxi H h J X K dx K = J XdAx = X(Ax). 

(This justifies using the same letter for both operators.) As usual, the value 
of the original problem © does not exceed the value of the dual problem ([7]): 
indeed, if x satisfies the constraints in (JSJ and A satisfies the constraint in ([7]), 

cx < (XA)x = X(Ax) = Xb. 

Now we have all components for the proof of the inequality < in Lemma [1] 

Proof of the inequality < in Lemma [7J It suffices to prove that the value of the 
dual problem (UJ does not exceed 2s/ K. Define A : K[0, 1] — !> [0, oo) by Xk(u) := 
(2/K - u/s) + for all k e {1, ... , K}, where t + is t if t > and otherwise. (In 
other words, assuming s < K/2, Xk : [0, 1] — > [0, oo) is the function with the 
subgraph of the smallest area among all functions that are linear when positive 
and whose graph passes through (s/K, 1/K).) Since 



Xb 



J X l + ••• + J X K < 2s/ K 



(with = in place of the last < when s < K/2), it remains to prove that the 
constraint in (J7J is satisfied. This is accomplished by the following chain of 
inequalities: 




K s I ~ \^\K s 

' \k=i v 

(2-(uiH Yu K )/s) + > ± 2 -(u 1 +-+u K )/s>i = l«i+-+u K <» = c(ui, . • . , u K ). 

□ 



3 Case K = 2 

In the case K = 2 upper copular probability admits a simple characterization. 

Lemma 2. If a nonempty Borel set E C [0, l] 2 is decreasing, its upper copular 
probability is 

C(E) = min (inf { Ul + u 2 \ («i,u 3 ) £ [0, l] 2 \ E) , l) . (8) 

Lemma [3 implies that the factor 2 in Q is optimal for K = 2: indeed, it shows 
that the function M a (pi,p 2 ) := ot(j>\ +p 2 ), where a > 0, satisfies C(M a < e) = 
min(e/a, 1) for all e £ [0, 1]; therefore, M a is a merging function if and only if 
a > 1. It is clear that Mi = M is the only precise merging function among M a . 
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Proof of Lemma\^ Let E be a nonempty decreasing Borel set in [0, l] 2 ; suppose 
C(E) is strictly less than the right-hand side of (|8]). Let t be any number strictly 
between C(E) and the right-hand side of ©. The copular probability measure 
concentrated on 

[(t,0),(o,t)]u[(M),(M)] 

has a value of at least t on E since E contains [(t, 0), (0, t)} . Therefore, C(E) > t. 
This contradiction proves the inequality > in 

The inequality < in ((5J) follows from Lemma [T] (part <). Indeed, denoting 
the right-hand side of (J5J) as s and assuming s < 1 (the case s = 1 is trivial), we 
have i? C E S+<L for an arbitrarily small e > 0, in the notation of ([5]). Therefore, 
by Lemma[Il C(E) < C(E s+e ) < s + e. □ 

A merging function F\ dominates a merging function Fi if F± < F%- The 
following corollary of Lemma [2] says that, in the case K = 2, the merging 
function Q is dominated by all precise merging functions. This is not true when 
K > 2: for example, for the Bonfcrroni function (p} we have M(p, . . . ,p) = 2p < 
Kp = F(p,...,p), 

Corollary 1. When K = 2, any precise merging function dominates M . 

Proof. Let F : [0, l] 2 — > [0, oo) be a merging function that does not dominate 
M. Choose (u 1 ,U2) S [0, l] 2 such that F(ui,U2) > u\ + u 2 and choose e € 
(ui + u-2, F{u\, U2). Since {F < e} does not contain (ui,v,2), we have C(F < 
e) < u\ + U2 < e, and so F is not precise. □ 
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